Since the early 1990's, the number of people using the World Wide Web and the Internet has grown at a substantial rate. As more users take advantage of the services available on the Internet by registering on websites, posting comments and information electronically, or simply interacting with companies that post information about others (such as online newspapers), more and more information about the users is available. There is also a substantial amount of information available in publicly and privately available databases, such as LEXISNEXIS. Sending a query to the one or more of the above resources, using the name of a person or entity and other identifying information, can return highly dimensional data sets that occupy large amounts of memory. The large data sets can consume excessive system resources to process or can even be large enough that it is not feasible to contain the data set in virtual memory.
Software developed to process queries in such systems can consist of large amounts of code that can exist in a single file. The file can be extremely large, containing thousands of line of code, which can make modification of such code difficult. Additionally, software structures within the file can involve synchronous or asynchronous execution of a particular job. In particular, for asynchronous execution of the job, the identification and modification of these structures can be problematic.
Additionally, flow control programming structures often require normalization of files so they can be passed to different machines. This normalization process can act as significant overhead for the system, especially, when large numbers of files are being passed.